Scaling Law for Recommendation Models: Towards General-purpose User Representations

Recent advancement of large-scale pretrained models such as BERT, GPT-3, CLIP, and Gopher, has shown astonishing achievements across various task domains. Unlike vision recognition and language models, studies on general-purpose user representation at scale still remain underexplored. Here we explore the possibility of general-purpose user representation learning by training a universal user encoder at large scales. We demonstrate that the scaling law is present in user representation learning areas, where the training error scales as a power-law with the amount of computation. Our Contrastive Learning User Encoder (CLUE), optimizes task-agnostic objectives, and the resulting user embeddings stretch our expectation of what is possible to do in various downstream tasks. CLUE also shows great transferability to other domains and companies, as performances on an online experiment shows significant improvements in Click-Through-Rate (CTR). Furthermore, we also investigate how the model performance is influenced by the scale factors, such as training data size, model capacity, sequence length, and batch size. Finally, we discuss the broader impacts of CLUE in general.Comment: Accepted at AAAI 2023. This version includes the technical appendi

Deformable Graph Transformer

Transformer-based models have recently shown success in representation learning on graph-structured data beyond natural language processing and computer vision. However, the success is limited to small-scale graphs due to the drawbacks of full dot-product attention on graphs such as the quadratic complexity with respect to the number of nodes and message aggregation from enormous irrelevant nodes. To address these issues, we propose Deformable Graph Transformer (DGT) that performs sparse attention via dynamically sampled relevant nodes for efficiently handling large-scale graphs with a linear complexity in the number of nodes. Specifically, our framework first constructs multiple node sequences with various criteria to consider both structural and semantic proximity. Then, combining with our learnable Katz Positional Encodings, the sparse attention is applied to the node sequences for learning node representations with a significantly reduced computational cost. Extensive experiments demonstrate that our DGT achieves state-of-the-art performance on 7 graph benchmark datasets with 2.5 - 449 times less computational cost compared to transformer-based graph models with full attention.Comment: 16 pages, 3 figure

Pivotal Role of Language Modeling in Recommender Systems: Enriching Task-specific and Task-agnostic Representation Learning

Recent studies have proposed unified user modeling frameworks that leverage user behavior data from various applications. Many of them benefit from utilizing users' behavior sequences as plain texts, representing rich information in any domain or system without losing generality. Hence, a question arises: Can language modeling for user history corpus help improve recommender systems? While its versatile usability has been widely investigated in many domains, its applications to recommender systems still remain underexplored. We show that language modeling applied directly to task-specific user histories achieves excellent results on diverse recommendation tasks. Also, leveraging additional task-agnostic user histories delivers significant performance benefits. We further demonstrate that our approach can provide promising transfer learning capabilities for a broad spectrum of real-world recommender systems, even on unseen domains and services.Comment: 14 pages, 5 figures, 9 table

Developmental endothelial locus-1 as a potential biomarker for the incidence of acute exacerbation in patients with chronic obstructive pulmonary disease

Background Despite the high disease burden of chronic obstructive pulmonary disease (COPD) and risk of acute COPD exacerbation, few COPD biomarkers are available. As developmental endothelial locus-1 (DEL-1) has been proposed to possess beneficial effects, including anti-inflammatory effects, we hypothesized that DEL-1 could be a blood biomarker for COPD. Objective To elucidate the role of plasma DEL-1 as a biomarker of COPD in terms of pathogenesis and for predicting acute exacerbation. Methods Cigarette smoke extract (CSE) or saline was intratracheally administered to wild-type (WT) and DEL-1 knockout (KO) C57BL/6 mice. Subsequently, lung sections were obtained to quantify the degree of emphysema using the mean linear intercept (MLI). Additionally, plasma DEL-1 levels were compared between COPD and non-COPD participants recruited in ongoing prospective cohorts. Using negative binomial regression analysis, the association between the plasma DEL-1 level and subsequent acute exacerbation risk was evaluated in patients with COPD. Results In the in vivo study, DEL-1 KO induced emphysema (KO saline vs. WT saline; P = 0.003) and augmented CSE-induced emphysema (KO CSE vs. WT CSE; P < 0.001) in 29 mice. Among 537 participants, patients with COPD presented plasma log (DEL-1) levels lower than non-COPD participants (P = 0.04), especially non-COPD never smokers (P = 0.019). During 1.2 ± 0.3 years, patients with COPD in the lowest quartile of Log(DEL-1) demonstrated an increased risk of subsequent acute exacerbation, compared with those in the highest quartile of Log(DEL-1) (adjusted incidence rate ratio, 3.64; 95% confidence interval, 1.03–12.9). Conclusion Low DEL-1 levels are associated with COPD development and increased risk of subsequent COPD acute exacerbation. DEL-1 can be a useful biomarker in patients with COPD.This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2019R1C1C1007918). This research was also supported by funds (2016ER670100, 2016ER670101, 2016ER670102 and 2018ER670100, 2018ER670101, 2018ER670102) from Research of Korea Centers for Disease Control and Prevention

The IPIN 2019 Indoor Localisation Competition—Description and Results

IPIN 2019 Competition, sixth in a series of IPIN competitions, was held at the CNR Research Area of Pisa (IT), integrated into the program of the IPIN 2019 Conference. It included two on-site real-time Tracks and three off-site Tracks. The four Tracks presented in this paper were set in the same environment, made of two buildings close together for a total usable area of 1000 m 2 outdoors and and 6000 m 2 indoors over three floors, with a total path length exceeding 500 m. IPIN competitions, based on the EvAAL framework, have aimed at comparing the accuracy performance of personal positioning systems in fair and realistic conditions: past editions of the competition were carried in big conference settings, university campuses and a shopping mall. Positioning accuracy is computed while the person carrying the system under test walks at normal walking speed, uses lifts and goes up and down stairs or briefly stops at given points. Results presented here are a showcase of state-of-the-art systems tested side by side in real-world settings as part of the on-site real-time competition Tracks. Results for off-site Tracks allow a detailed and reproducible comparison of the most recent positioning and tracking algorithms in the same environment as the on-site Tracks

1H NMR-Based Metabolomic Analysis of Intraerythrocytic Stages of P. falciparum and Antimalarial Treatments

1H NMR provides an indirect approach that can be used to study the metabolism of the malaria parasite, Plasmodium falciparum, in human erythrocytes. It allows for facile analysis of the metabolites that the parasite excretes into and consumes from the surrounding media to gain insight into this dynamic system. This study carefully processes a very large batch of NMR data using MestReNova through a rigorous procedure. This is followed by multivariate statistical analyses in SIMCA to metabolically understand how erythrocytes respond to parasitic infection and subsequent treatment with artemisinin, chloroquine, or a candidate antimalarial, tiamulin. Trajectories reveal separation between the infected and artemisinin sets, and the uninfected, chloroquine, and tiamulin sets. Coefficient plots provide an introduction to the specific metabolites, the concentrations of which change systematically in the media. This study conclusively demonstrates that tiamulin treatment results in highly similar metabolic effects to those of chloroquine, both in terms of overall time-evolution trajectories and metabolic exchange with the medium during each individual phase. It also appears to normalize cells to be similar to control uninfected cells. This cost effective and non-targeted 1H NMR approach to profile metabolites can be adopted for large-scale drug screenings

HSP70-Homolog DnaK of Pseudomonas aeruginosa Increases the Production of IL-27 through Expression of EBI3 via TLR4-Dependent NF-κB and TLR4-Independent Akt Signaling

IL-27, a heterodimeric cytokine composed of the p28 subunit and Epstein&ndash;Barr virus-induced gene 3 (EBI3), acts as a potent immunosuppressant and thus limits pathogenic inflammatory responses. IL-27 is upregulated upon Pseudomonas aeruginosa infection in septic mice, increasing susceptibility to the infection and decreasing clearance of the pathogen. However, it remains unclear which P. aeruginosa-derived molecules promote production of IL-27. In this study, we explored the mechanism by which P. aeruginosa DnaK, a heat shock protein 70-like protein, induces EBI3 expression, thereby promoting production of IL-27. Upregulation of EBI3 expression did not lead to an increase in IL-35, which consists of the p35 subunit and EBI3. The IL-27 production in response to DnaK was biologically active, as reflected by stimulation of IL-10 production. DnaK-mediated expression of EBI3 was driven by two distinct signaling pathways, NF-&kappa;B and Akt. However, NF-&kappa;B is linked to TLR4-associated signaling pathways, whereas Akt is not. Taken together, our results reveal that P. aeruginosa DnaK potently upregulates EBI3 expression, which in turn drives production of the prominent anti-inflammatory cytokine IL-27, as a consequence of TLR4-dependent activation of NF-&kappa;B and TLR4-independent activation of the Akt signaling pathway

Synthesis of discrete bottlebrush polymers via the iterative convergent growth technique and post-functionalization

Bottlebrush polymers (BBPs) have unique physical properties that arise from polymeric side chains that are densely grafted onto their polymer backbone. Despite their promising attributes, well-defined BBPs are challenging to prepare partly due to steric hindrance encountered during macromonomer polymerization or during the coupling of side-chain polymers with functional groups present in the repeating units of the backbone polymers. Here we report the synthesis of perfectly discrete BBPs with absolutely defined chemical structures. The copolyester backbone, composed of up to 64 repeating units of an alternating sequence of lactic acid and 2-hydroxy-4-pentenoic acid, was synthesized using an iterative convergent growth technique. Discrete side chains, such as discrete polylactides (LA(n)-SH), were introduced in the presence of the vinyl groups of the 2-hydroxy-4-pentenoic acid residues on the copolyester backbone using thiol-ene click chemistry promoted by UV light (lambda = 365 nm), which produced a fully grafted BBP, t-LP(LA(8))32-LA(8) (Molecular weight (MW = 34 kDa). We also synthesized an amphiphilic bottlebrush block copolymer (BBCP) (MW = 19 kDa) by introducing the hydrophobic dodecanethiol and discrete poly(ethylene glycol) as side chains. The resulting BBCP self-assembled into well-defined micelles in water. Discrete BBCPs may serve as model systems that help to elucidate the role of the branched architecture on the physical and chemical behavior of BBCPs in bulk and solution.N

Stretchable High-Resolution User-Interactive Synesthesia Displays for Visual-Acoustic Encryption

Multifunctional displays, which have various functions in single-device systems without external circuits, are actively investigated as future human-machine interfaces owing to performability of unprecedented functions in compact design. However, their application is limited to visualize the mechanical/electrical signals in light. Herein, stretchable high-resolution multicolor synesthesia display, which can generate synchronized sound and light as input/output sources, is presented by transfer-printing. Transfer-printed emissive composite leads to display with enhanced optical performance and fine sound pressure level. Owing to inherent stretchability of the device, the synesthesia display can stably operate under static and dynamic deformation without distortion in sound relative to the input waveform. User-interactive synesthesia displays are demonstrated for visual-acoustic encryption, which facilitate advanced encryption, as well as multiplex quick response code that bridges multiple domains with a single device. This approach provides new directions for multifunctional displays, with potential applications in reinforced authentication

Mosquito-specific microRNA-1174 targets serine hydroxymethyltransferase to control key functions in the gut

Lineage-specific microRNAs (miRNAs) may contribute to functions specific to hematophagous mosquitoes and, as such, have potential for contributing to the development of future mosquito control approaches. Here we report that the mosquito- and gut-specific miRNA, miR-1174, is required for proper sugar absorption, fluid excretion, blood intake, and, consequently, egg maturation and survival in female mosquitoes. miR-1174 is highly expressed and localized in the posterior midgut, the blood-digesting portion of the mosquito alimentary canal. Depletion of miR-1174 results in severe defects in sugar absorption and blood intake. We identified serine hydroxymethyltransferase (SHMT) is a direct miR-1174 target. The adverse phenotypes caused by miR-1174 silencing were rescued by SHMT RNA interference. Our results suggest that miR-1174 is essential for fine-tuning the SHMT transcript to levels necessary for normal mosquito gut functions